Imaging systems with clear filter pixels

ABSTRACT

An image sensor may have an array of image sensor pixels arranged in color filter unit cells each having one red image pixel that generates red image signals, one blue image pixel that generate blue image signals, and two clear image sensor pixels that generate white image signals. The image sensor may be coupled to processing circuitry that performs filtering operations on the red, blue, and white image signals to increase noise correlations in the image signals that reduce noise amplification when applying a color correction matrix to the image signals. The processing circuitry may extract a green image signal from the white image signal. The processing circuitry may compute a scaling value that includes a linear combination of the red, blue, white and green image signals. The scaling value may be applied to the red, blue, and green image signals to produce corrected image signals having improved image quality.

This application is a continuation of patent application Ser. No.15/343,425, filed Nov. 4, 2016, which is a continuation of patentapplication Ser. No. 14/871,520, filed Sep. 30, 2015, which is acontinuation of patent application Ser. No. 13/736,768, filed Jan. 8,2013, which claims the benefit of provisional patent application No.61/612,819, filed Mar. 19, 2012, which are hereby incorporated byreference herein in their entireties. This application claims thebenefit of and claims priority to patent application Ser. No.15/343,425, filed Nov. 4, 2016, patent application Ser. No. 14/871,520,filed Sep. 30, 2015, patent application Ser. No. 13/736,768, filed Jan.8, 2013, and provisional patent application No. 61/612,819, filed Mar.19, 2012.

BACKGROUND

This relates generally to imaging devices, and more particularly, toimaging devices with clear image pixels.

Image sensors are commonly used in electronic devices such as cellulartelephones, cameras, and computers to capture images. In a typicalarrangement, an electronic device is provided with an array of imagepixels arranged in pixel rows and pixel columns. Circuitry is commonlycoupled to each pixel column for reading out image signals from theimage pixels.

Conventional imaging systems employ a single image sensor in which thevisible light spectrum is sampled by red, green, and blue (RGB) imagepixels arranged in a Bayer mosaic pattern. The Bayer Mosaic patternconsists of a repeating cell of two-by-two image pixels, with two greenpixels diagonally opposite one another, and the other corners being redand blue. However, the Bayer pattern does not readily enable furtherminiaturization of image sensors via smaller image pixel sizes becauseof limitations of signal to noise ratio (SNR) in the image signalscaptured from the image pixels.

One means of improving SNR is to increase the available image signal byincreasing light exposure at low light levels, where SNR limits theimage quality. One conventional method is the use of subtractivefilters, in which, for example, red, green, and blue image pixels arereplaced by cyan, magenta, and yellow image pixels. However, thesesignals must generally be converted to RGB or some equivalent outputimage signal colors to be able to drive most conventional imagedisplays. This transformation generally involves the modification ofcaptured image signals using a color correction matrix (CCM), which canamplify noise, so that the effect of the exposure increase iscompromised.

It would therefore be desirable to be able to provide imaging deviceswith improved means of capturing and processing image signals.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an illustrative electronic device having animaging system in accordance with an embodiment of the presentinvention.

FIG. 2 is a diagram of an illustrative pixel array and associatedcontrol circuitry for reading out pixel data from image pixels alongcolumn lines in an image sensor in accordance with an embodiment of thepresent invention.

FIGS. 3-5 are diagrams of illustrative pixel unit cells having clearfilter pixels in accordance with embodiments of the present invention.

FIG. 6 is a flow chart of illustrative steps that may be performed byprocessing circuitry in an imaging system to process image signalsreceived from a filtered pixel array in accordance with an embodiment ofthe present invention.

FIG. 7 is a flow chart of illustrative steps that may be performed byprocessing circuitry in an imaging system to demosaic and filter imagesignals received from a filtered pixel array in accordance with anembodiment of the present invention.

FIG. 8 is a flow chart of illustrative steps that may be performed byprocessing circuitry in an imaging system to apply a point filter toimage signals received from a filtered pixel array in accordance with anembodiment of the present invention.

FIG. 9 is a block diagram of a processor system employing the embodimentof FIG. 1 in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

Electronic devices such as digital cameras, computers, cellulartelephones, and other electronic devices include image sensors thatgather incoming light to capture an image. The image sensors may includearrays of image pixels. The pixels in the image sensors may includephotosensitive elements such as photodiodes that convert the incominglight into image signals. Image sensors may have any number of pixels(e.g., hundreds or thousands or more). A typical image sensor may, forexample, have hundreds of thousands or millions of pixels (e.g.,megapixels). Image sensors may include control circuitry such ascircuitry for operating the image pixels and readout circuitry forreading out image signals corresponding to the electric charge generatedby the photosensitive elements. Readout circuitry may include selectablereadout circuitry coupled to each column of pixels that can be enabledor disabled to reduce power consumption in the device and improve pixelreadout operations.

FIG. 1 is a diagram of an illustrative electronic device that uses animage sensor to capture images. Electronic device 10 of FIG. 1 may be aportable electronic device such as a camera, a cellular telephone, avideo camera, or other imaging device that captures digital image data.Camera module 12 may be used to convert incoming light into digitalimage data. Camera module 12 may include one or more lenses 14 and oneor more corresponding image sensors 16. During image capture operations,light from a scene may be focused onto image sensor 16 by lens 14. Imagesensor 16 may include circuitry for converting analog pixel data intocorresponding digital image data to be provided to processing circuitry18. If desired, camera module 12 may be provided with an array of lenses14 and an array of corresponding image sensors 16.

Processing circuitry 18 may include one or more integrated circuits(e.g., image processing circuits, microprocessors, storage devices suchas random-access memory and non-volatile memory, etc.) and may beimplemented using components that are separate from camera module 12and/or that form part of camera module 12 (e.g., circuits that form partof an integrated circuit that includes image sensors 16 or an integratedcircuit within module 12 that is associated with image sensors 16).Image data that has been captured by camera module 12 may be processedand stored using processing circuitry 18. Processed image data may, ifdesired, be provided to external equipment (e.g., a computer or otherdevice) using wired and/or wireless communications paths coupled toprocessing circuitry 18.

As shown in FIG. 2, image sensor 16 may include a pixel array 200containing image sensor pixels 190 (sometimes referred to herein asimage pixels 190) and control and processing circuitry 122. Array 200may contain, for example, hundreds or thousands of rows and columns ofimage sensor pixels 190. Control circuitry 122 may be coupled to rowdecoder circuitry 124 and column decoder circuitry 126. Row decodercircuitry 124 may receive row addresses from control circuitry 122 andsupply corresponding row control signals such as reset, row-select,transfer, and read control signals to pixels 190 over control paths 128.One or more conductive lines such as column lines 40 may be coupled toeach column of pixels 190 in array 200. Column lines 40 may be used forreading out image signals from pixels 190 and for supplying bias signals(e.g., bias currents or bias voltages) to pixels 190. During pixelreadout operations, a pixel row in array 200 may be selected using rowdecoder circuitry 124 and image data associated with image pixels 190 inthat pixel row can be read out along column lines 40.

Column decoder circuitry 126 may include sample-and-hold circuitry,amplifier circuitry, analog-to-digital conversion circuitry, biascircuitry, column memory, latch circuitry for selectively enabling ordisabling the column circuitry, or other circuitry that is coupled toone or more columns of pixels in array 200 for operating pixels 190 andfor reading out image signals from pixels 190. Column decoder circuitry126 may be used to selectively provide power to column circuitry on aselected subset of column lines 40. Readout circuitry such as signalprocessing circuitry associated with column decoder circuitry 126 (e.g.,sample-and-hold circuitry and analog-to-digital conversion circuitry)may be used to supply digital image data to processor 18 (FIG. 1) overpath 210 for pixels in chosen pixel columns.

Image sensor pixels such as image pixels 190 are conventionally providedwith a color filter array which allows a single image sensor to samplered, green, and blue (RGB) light using corresponding red, green, andblue image sensor pixels arranged in a Bayer mosaic pattern. The Bayermosaic pattern consists of a repeating unit cell of two-by-two imagepixels, with two green image pixels diagonally opposite one another andadjacent to a red image pixel diagonally opposite to a blue image pixel.However, limitations of signal to noise ratio (SNR) that are associatedwith the Bayer Mosaic pattern make it difficult to reduce the size ofimage sensors such as image sensor 16. It may therefore be desirable tobe able to provide image sensors with an improved means of capturingimages.

In one suitable example that is sometimes discussed herein as anexample, the green pixels in a Bayer pattern are replaced by clear imagepixels as shown in FIG. 3. As shown in FIG. 3, a unit cell 192 of imagepixels 190 may be formed from two clear image pixels (sometimes referredto herein as white (W) image pixels) that are diagonally opposite oneanother and adjacent to a red (R) image pixel that is diagonallyopposite to a blue (B) image pixel. White image pixels 190 in unit cell192 may be formed with a visibly transparent color filter that transmitslight across the visible light spectrum (e.g., white pixels 190 cancapture white light). Clear image pixels 190 may have a naturalsensitivity defined by the material that forms the transparent colorfilter and/or the material that forms the image sensor pixel (e.g.,silicon). The sensitivity of clear image pixels 190 may, if desired, beadjusted for better color reproduction and/or noise characteristicsthrough use of light absorbers such as pigments. Unit cell 192 may berepeated across image pixel array 200 to form a mosaic of red, white,and blue image pixels 190. In this way, red image pixels may generatered image signals in response to red light, blue image pixels maygenerate blue image signals in response to blue light, and white imagepixels may generate white image signals in response to white light. Thewhite image signals may also be generated by the white image pixels inresponse to any suitable combination of red, blue, and/or green light.

The unit cell 192 of FIG. 3 is merely illustrative. If desired, anycolor image pixels may be formed adjacent to the diagonally opposingwhite image pixels in unit cell 192. For example, a unit cell 194 may bedefined by two white image pixels 190 that are formed diagonallyopposite one another and adjacent to a red image pixel that isdiagonally opposite to a green (G) image pixel, as shown in FIG. 4. Inyet another suitable arrangement, a unit cell 196 may be defined by twowhite image pixels 190 that are formed diagonally opposite one anotherand adjacent to a blue image pixel that is diagonally opposite to agreen image pixel, as shown in the example of FIG. 5.

White image pixels W can help increase the signal-to-noise ration (SNR)of image signals captured by image pixels 190 by gathering additionallight in comparison with image pixels having a narrower color filter(e.g., a filter that transmits light over a subset of the visible lightspectrum), such as green image pixels. White image pixels W mayparticularly improve SNR in low light conditions in which the SNR cansometimes limit the image quality of images. Image signals gathered fromimage pixel array 200 having white image pixels (e.g., as shown in FIGS.3-5) may be converted to red, green, and blue image signals to becompatible with circuitry and software that is used to drive most imagedisplays (e.g., display screens, monitors, etc.). This conversiongenerally involves the modification of captured image signals using acolor correction matrix (CCM). If care is not taken, color correctionoperations can undesirably amplify noise.

In one suitable arrangement, noise generated by the CCM may be reducedby implementing strong de-noising (e.g., chroma de-noising) prior toapplying the CCM to gathered image signals. Chroma de-noising may beperformed by processing circuitry 18 (FIG. 1) by applying a chromafilter to image signals gathered by image pixels 190. The chroma filtermay serve to increase noise correlation between image signals fromdifferent colored image pixels (e.g., red, white, and blue imagesignals). Increasing noise correlation between image signals fromdifferent colored image pixels may reduce noise amplification by theCCM, leading to improved final image quality. In another arrangement,noise amplified by the CCM may be compensated for by applying aso-called “point filter” to the captured image signals. The point filtermay use high fidelity white image signals to enhance the quality of red,green, and blue image signals produced using the CCM. If desired, imagesensor 16 may implement both chroma de-noising and the point filter toreduce noise amplification by the CCM to yield improved luminanceperformance in the final image.

FIG. 6 shows a flow chart of illustrative steps that may be performed byprocessing circuitry such as processing circuitry 18 of FIG. 1 toprocess image signals gathered by a filtered pixel array such as pixelarray 200 (e.g., a pixel array that is free of green image pixels). Thesteps of FIG. 6 may, for example, be performed by processing circuitry18 to reduce noise in image signals captured using unit cells havingwhite image pixels such as those shown in FIGS. 3-5.

At step 100, image sensor 16 may capture image signals from a scene. Theimage signals captured by image sensor 16 may include white imagesignals generated in response to light gathered with the white pixels.If desired, the image signals may also include one or more of red imagesignals, blue image signals, or green image signals depending on theconfiguration of image pixels used (e.g., if unit cell 192 of FIG. 3 isused then the image signals may include red, white, and blue imagesignals, if unit cell 194 of FIG. 4 is used then the image signals mayinclude red, white, and green image signals, etc.). In the example ofFIG. 6, red (R′), white (W′), and blue (B′) image signals may becaptured. The red image signals may have a first spectral response value(an integrated signal power level as a function of the frequency oflight received by red image sensor pixels), the blue image signals mayhave a second spectral response value, and the white image signals mayhave a third spectral response value that is, for example, greater thanseventy five percent of a sum of the first and second spectral responsevalues (e.g., white image signals having a broad sensitivity for anequal energy radiator over the visible light spectrum with standard CIEilluminant E). The image signals may have image values corresponding tolight captured by each image pixel 190 (e.g., red image signals mayinclude a red image value, blue image signals may include a blue imagevalue, etc.). The captured image signals may be conveyed to processingcircuitry 18 for image processing.

At step 102, a white balance operation may be performed on the capturedimage signals. In the example of FIG. 6, a white-balanced red imagesignal (R), white-balanced white image signal (W), and white-balancedblue image signal (B) may be produced.

At step 104, processing circuitry 18 may demosaic and apply a chromafilter to the white-balanced image signals to extract red, white, andblue image data from the white-balanced image signals. The chroma filtermay be applied to chroma de-noise the white-balanced image signals.Processing circuitry 18 may, for example, demosaic the image signals andapply the chroma filter simultaneously, sequentially, or in aninterspersed manner. This process of applying a chroma filter anddemosaicking the image signals may sometimes be referred to herein as“chroma demosaicking.” The chroma filter may increase noise correlationbetween image signals of each color (e.g., noise fluctuations in thered, white, and blue channels may increase or decrease together in acorrelated manner). For example, processing circuitry 18 may increasethe correlated noise between the red, white, and green image signals toas much as 70% or more of all noise associated with the red, white, andgreen image signals.

By increasing noise correlation, processing circuitry 18 may reduce theamount of noise amplification generated when a CCM is applied to theimage signals. Chroma demosaicking the image signals may allow missingcolor image signals (e.g., image signals of colors not generated by theimage pixels) to be determined from available color image signals. Inthis example, green image signals may be missing from the gathered imagesignals because no green color filter is used in unit cell 192 (FIG. 3).A green image signal may be determined using the white, red, and blueimage signals (e.g., by performing subtraction operations). In general,any of the primary additive colors (e.g., red, green, and blue) may bedetermined using the available color image signals. It may be desirableto produce red, green, and blue image signals regardless of the colorfilters used on image pixel array 200 because display systems oftendisplay images using red, green, and blue pixels.

At step 106, processing circuitry 18 may apply a color correction matrix(CCM) to the red image data, white image data, and blue image data. TheCCM may, for example, extract green image data from the white image datato generate red, green, and blue image data. For example, the CCM mayconvert the image data into standard red, standard green, and standardblue image data (sometimes referred to collectively as linear sRGB imagedata or simply sRGB image data). In another suitable arrangement, theCCM may extract green image data from the red and/or blue image data. Ifdesired, gamma correction processes may be performed on the linear sRGBimage data. After gamma correction, the sRGB image data may be used fordisplay using an image display device. In some cases, it may bedesirable to provide additional noise reduction (e.g., by applying apoint filter to the sRGB image data) to further mitigate the noiseamplification generated by applying the CCM to the red, white, and blueimage data. Processing circuitry 18 may preserve the white image datafor further processing of the sRGB image data during optional step 108.

At optional step 108, processing circuitry 18 may apply a point filterto the image data (e.g., to the sRGB image data produced after applyingthe CCM to the red, white, and blue image data). The point filter mayoperate on the sRGB image data to generate corrected sRGB data. Thepoint filter may serve to further reduce noise amplification caused byapplying the CCM to the red, white, and blue image data. When displayedusing a display system, the corrected sRGB data thereby provide betterimage quality (e.g., better luminance performance) when compared to thesRGB data prior to applying the point filter.

FIG. 7 shows a flow chart of illustrative steps that may be performed byprocessing circuitry 18 to demosaic and filter image signals receivedfrom image pixel array 200. The steps of FIG. 7 may, for example, beperformed by processing circuitry 18 to perform chroma demosaicking onred, white, and blue image signals gathered by image pixels 190 togenerate sufficient noise correlation in red, white, and blue imagedata. The steps of FIG. 7 may, for example, be performed as part of step104 of FIG. 6.

At step 110, processing circuitry 18 may demosaic the white image signalto produce white image data (e.g., a white image value for each imagepixel). In another suitable arrangement, white image values may beproduced for a combination of available image pixels 190. The whiteimage values may be used to compute difference values using the red andblue image signals to increase noise correlation between the red, white,and blue image signals.

At step 112, processing circuitry 18 may generate red difference valuesby subtracting the white image values from the red image values for eachpixel. Processing circuitry 18 may generate blue difference values bysubtracting the white image values from the blue image values. The reddifference values may, for example, be computed for each red image pixeland the blue difference values may be computed for each blue image pixelof image pixel array 200.

At step 114, processing circuitry 18 may filter the red differencevalues and the blue difference values using a chroma filter. The chromafilter may be applied to the red and blue difference values by, forexample, performing a weighted average of difference values computedover a kernel of image pixels 190 (e.g., a weighted average of a groupof difference values that were computed by performing step 112). Thekernel of image pixels may be defined as a subset of the image pixels inimage pixel array 200 over which the chroma filtering is being performed(e.g., the kernel may include some or all of the image pixels in imagepixel array 200). For example, when a 5 pixel by 5 pixel kernel is used,a weighted average of difference values is calculated for a 5 pixel by 5pixel subset of image pixels 190 in image pixel array 200 whenperforming chroma filtering (e.g., a weighted sum of difference valuesmay be computed for a given image pixel 190 using difference values at25 surrounding image pixels in image pixel array 200). In general, akernel of any desired size may be used.

At step 116, the white image values may be added to the chroma filteredred difference values and the chroma filtered blue difference values togenerate chroma filtered red image values and chroma filtered blue imagevalues, respectively.

At step 118, processing circuitry 18 may demosaic the chroma filteredred image values and the chroma filtered blue image values to producered image data and blue image data (e.g., red and blue image data thathas been chroma demosaicked) with increased correlated noise. Thedemosaicked white image data and the chroma demosaicked red and blueimage data may then be operated on using the CCM to generate standardred, standard green, and standard blue (sRGB) image data as describedabove in connection with step 106 of FIG. 6.

FIG. 7 is merely illustrative. If desired, processing circuitry 18 maydemosaic the chroma filtered red and blue image values prior togenerating the red and blue difference values (e.g., processingcircuitry 18 may perform step 118 prior to step 112).

If chroma filtering of the difference values is performed over asufficiently large kernel of image pixels 190, minimal noise from thered and blue image signals may remain in the red and blue differencevalues after chroma filtering (e.g., after performing step 114). Forexample, if the kernel has a size of 15 pixels by 15 pixels or greater,chroma filtering may reduce noise in the red and blue chroma filtereddifference values to negligible levels. If desired, the kernel of imagepixels 190 may include image pixels located in multiple image pixelarrays 200, image pixels located in multiple image sensors 16, and/orimage pixels used during multiple time frames (e.g., to allow fortemporal denoising). When the white image values are added to the chromafiltered difference values, noise in the white image values may dominateover noise in the difference values. In this way, noise in the red andblue image data produced at step 116 may be substantially equal to noisein the white image data. Noise in the red and blue image data maythereby be highly correlated, resulting in reduced noise amplificationby the CCM. This process may produce less noise amplification by the CCMthan when a Bayer pattern is used for image pixel array 200.

The CCM may operate on the red, white, and blue image data to producelinear sRGB data at step 106 (FIG. 6). For example, the CCM may extractinformation from the white image data to generate the standard greendata. The white image data (e.g., the demosaicked white image dataproduced at step 104) may be preserved after operating on the image datawith the CCM. The sRGB image data may be represented in otherthree-dimensional spaces such as a luminance-chroma-hue (LCH) space. Inan LCH space, the luminance channel (L) may be related to the brightnessof an image captured by image sensor 16, the chroma channel (C) may berelated to the color saturation of an image, and the hue channel may berelated to the specific color of the image (e.g., red, purple, yellow,green, etc.). The perception of noise and sharpness in a displayed imagemay be affected by noise and signal variations in the luminance channel.The SNR in the image data may be improved by transforming the sRGB datato LHC data, replacing a luminance value in the luminance channel with awhite image value (which correlates well with overall image brightnessdue to the broad spectrum of the white image signal), and transformingLHC data back to sRGB data. In this way, noise amplification caused bythe CCM may be suppressed in the luminance channel, where noise isparticularly noticeable to a viewer when viewing a displayed image.

As described above in connection with optional step 108 of FIG. 6, apoint filter may be applied to the linear sRGB data to produce correctedsRGB data using the white image data. The point filter may operate on asingle image pixel 190 without information from adjacent image pixels190, whereas chroma demosaicking may require image signals (e.g.,difference values) from multiple image pixels (e.g., a kernel of imagepixels) when being applied to image signals at a single image pixel 190.For example, the point filter may operate on a standard red value,standard green value, and standard blue value for each image pixel. Toperform point filter operations on the sRGB data, processing circuitry18 may use the red image data, white image data, and blue image data(e.g., the image data prior to applying the CCM) to compute an original(raw) luminance signal. The original luminance signal may be a linearcombination (e.g., a weighted sum) of the white image data, red imagedata, and blue image data. If desired, the white image data may beweighted more heavily than the red and blue image data in the linearcombination. Processing circuitry 18 may compute an implied luminancesignal that is a linear combination of the standard red, standard green,and standard blue image data (e.g., after applying the CCM to the imagedata). If desired, weights in the linear combination used to compute theimplied luminance signal may be substantially similar to the weightsused to compute the original luminance signal. The weights may beadjusted to modify the “strength” of the point filter (e.g., the degreeto which the point filter transforms or corrects the sRGB data).

Processing circuitry 18 may generate a scaling value (e.g., a scalingfactor to be applied to color corrected image values) by, in a simplestcase, dividing the original luminance signal by the implied luminancesignal. If desired, the scaling factor may include a numerator anddenominator. The numerator and/or the denominator of the scaling valuemay include a weighted sum of the original luminance signal and theimplied luminance signal. The scaling value may include adjustableweighting parameters that can be varied to adjust the strength of thepoint filter (e.g., the weighting parameters may be continuously variedto adjust the strength of the point filter from zero to a fullstrength). To apply the point filter to the sRGB data (e.g., to thestandard red, green, and blue image data), processing circuitry 18 maymultiply the sRGB data by the scaling value to produce the correctedsRGB data. For example, processing circuitry 18 may multiply thestandard red image data by the scaling value, the standard green imagedata by the scaling value, etc. If desired, the corrected sRGB data mayhave hue and chroma channels that are approximately preserved frombefore applying the point filter (e.g., upon conversion of the correctedsRGB data to LCH space). The corrected sRGB data may have improved noiseand/or sharpness due to inherited fidelity of the white image signals.

In a simplest case, the original luminance signal may be approximated bythe white image data. FIG. 8 shows a flow chart of illustrative stepsthat may be performed by processing circuitry 18 to apply a point filter(in the simplest case) to sRGB data after applying the CCM to the red,white, and blue image data (as an example). Processing circuitry 18 may,for example, apply the point filter to sRGB data for each image pixel190 in image pixel array 200. The steps of FIG. 8 may, for example, beperformed as part of step 108 of FIG. 6.

At step 130, processing circuitry 18 may generate an implied luminancevalue (e.g., a luminance value in LCH space) for a given image pixel 190by combining the red, green, blue image data (e.g., after applying aCCM). The implied luminance value may, for example, be computed as alinear combination of the red, green, and blue image data.

At step 132, processing circuitry 18 may generate a scaling value bydividing the white image values by the implied luminance value. Ifdesired, the scaling factor may be generated by dividing the white imagevalues by a weighted sum of the implied luminance value and the whiteimage value. The scaling factor may include adjustable weightingparameters that can be varied to adjust the strength of the point filter(e.g., the weighting parameters may be varied continuously to adjust thestrength of the point filter from zero to a full strength). The scalingvalue may, for example, be an operator that operates on the sRGB data.

At step 134, processing circuitry 18 may multiply the sRGB data by thescaling value to produce corrected sRGB data (e.g., corrected standardred, green, and blue image data). For example, processing circuitry 18may multiply the standard red image data by the scaling value, thestandard green image data by the scaling value, etc. The corrected sRGBdata may, if desired be provided to an image display. The corrected sRGBdata may have improved noise and/or sharpness when compared with thesRGB data prior to applying the point filter.

The examples of FIGS. 6-8 are merely illustrative. Any desired colorfilters may be used in conjunction with the white color filters shown inFIGS. 3-5 for obtaining color image signals. Any combination of desiredcolor filters may be used (e.g., any combination of red filters, greenfilters, cyan filters, infrared filters, ultraviolet filters, bluefilters, yellow filters, magenta filters, purple filters, etc.). Ifdesired, any other suitable three-dimensional spaces may be used forperforming the point filter operation.

If desired, any number of image pixel arrays 200 formed on any number ofimage sensors 16 may be used to capture images. Each image pixel arrayused may, for example, be used for image signals of a different color.For example, a first image pixel array may have clear filters forgenerating white image signals, a second image pixel array may have redfilters for generating red image signals, and a third image pixel arraymay have a blue filter for generating blue image signals. Image signalsfrom each of these arrays may be chroma demosaicked and/or operated onusing a point filter. Each image pixel array may, if desired, be formedon a different image sensor in device 10 such as image sensor 16 (e.g.,multiple image sensors 16 may be formed in device 10). Such anembodiment may, for example, allow for a shorter camera focal length anda thinner camera module.

FIG. 9 shows in simplified form a typical processor system 300, such asa digital camera, which includes an imaging device 2000 (e.g., animaging device 2000 such as imaging sensor 16 of FIGS. 1-8 employingclear color filters and the techniques for operations described above).The processor system 300 is exemplary of a system having digitalcircuits that could include imaging device 2000. Without being limiting,such a system could include a computer system, still or video camerasystem, scanner, machine vision, vehicle navigation, video phone,surveillance system, auto focus system, star tracker system, motiondetection system, image stabilization system, and other systemsemploying an imaging device.

The processor system 300 generally includes a lens 396 for focusing animage on pixel array 200 of device 2000 when a shutter release button397 is pressed, central processing unit (CPU) 395, such as amicroprocessor which controls camera and one or more image flowfunctions, which communicates with one or more input/output (I/O)devices 391 over a bus 393. Imaging device 2000 also communicates withthe CPU 395 over bus 393. The system 300 also includes random accessmemory (RAM) 392 and can include removable memory 394, such as flashmemory, which also communicates with CPU 395 over the bus 393. Imagingdevice 2000 may be combined with the CPU, with or without memory storageon a single integrated circuit or on a different chip. Although bus 393is illustrated as a single bus, it may be one or more busses or bridgesor other communication paths used to interconnect the system components.

Various embodiments have been described illustrating image sensorshaving clear image pixel filters and image processing techniques (e.g.,chroma demosaicking, applying a point filter, etc.) for reducing noisein image signals produced by the image signals.

An image sensor may have an array of image sensor pixels including redimage pixels that generate red image signals in response to red light,blue image pixels that generate blue image signals in response to bluelight, and clear image sensor pixels that generate white image signalsin response to at least red light, green light, and blue light (e.g.,white light). The image pixels may be arranged in an array of pixel unitcells each including a number of image pixels of different colors. Theimage sensor may be coupled to processing circuitry that performsfiltering operations on the red, blue, and white image signals toincrease noise correlations associated with the red, blue, and whiteimage signals. The processing circuitry may perform filtering operationsfor a given image pixel by, for example, generating a weighted sum ofimage signals generated by at least 25 image pixels in the image pixelarray. The weighted sum may include adjustable weights (e.g., weightsthat are adjusted based on observed image features). The weighted summay be generated for image signals captured during multiple time framesor from multiple image sensors. By generating the weighted sum formultiple time frames, the processing circuitry may reduce the size ofthe kernel of image pixels while successfully reducing image signalnoise.

This example is merely exemplary. In general, the array of image sensorpixels may include image sensor pixels of any desired colors (e.g.,image sensor pixels responsive to any color of light). For example, thearray of image sensor pixels may include a first group of image sensorpixels responsive to light of a first color, a second group of imagesensor pixels responsive to light of a second color, and a third groupof image sensor pixels responsive to light of a third color (e.g., red,blue, and white light). The first image signals may have a firstspectral response level (e.g., an integrated signal power level as afunction of the frequency of light received by the first group of imagesensor pixels), the second image signals may have a second spectralresponse level (e.g., an integrated signal power level as a function ofthe frequency of light received by the second group of image sensorpixels), and the third image signals may have a third spectral responselevel (e.g., an integrated signal power level as a function of thefrequency of light received by the third group of image sensor pixels).The third image signals may have a spectral response level that isgreater than the first and second spectral response levels (e.g., thethird spectral response level may be greater than 75 percent of the sumof the first and second spectral response levels). In other words, thethird image signals may be captured in response to a broader range oflight frequencies than the first and second image signals.

The processing circuitry may, if desired, generate an estimatedluminance value (e.g., a luminance value in LCH space) using the first,second, and third image signals. The processing circuitry may generatetransformed first, second, and third image signals by transforming thefirst, second, and third image signals into a derived trichromatic space(e.g., a linear sRGB space, CIE space, XYZ space, Bayer space, etc.).The processing circuitry may, for example, generate the transformedfirst, second, and third image signals by performing a linearcombination of the first, second, and third image signals. Theprocessing circuitry may generate a derived luminance value (e.g., aluminance value in an LCH space) by combining the transformed first,second, and third image signals. The processing circuitry may comparethe derived luminance value with the estimated luminance value andmodify the transformed first, second, and third image signals so thatthe derived luminance value approaches the estimated luminance value(e.g., so that the derived luminance value sufficiently matches theestimated luminance value).

The processing circuitry may, if desired, process image data includingfirst image signals of a first color, second image signals of a secondcolor that is different from the first color, and white image signalsusing an image sensor having processing circuitry. The processingcircuitry may generate third image signals of a third color that isdifferent from the first and second colors using the white imagesignals. The processing circuitry may combine the first image signals,the second image signals, and the third image signals to form a derivedluminance value, and may compute an estimated luminance value from thefirst color image signals, the second color image signals, and the whiteimage signals. The processing circuitry may form the derived luminancevalue by combining the white image signals with the first, second, andthird image signals.

The processing circuitry may modify the first image signals, the secondimage signals, and the third image signals using the derived luminancevalue and the estimated luminance value. For example, the processingcircuitry may compute a scaling value based on the derived luminancevalue and the estimated luminance value and may multiply the first,second, and third image signals by the generated scaling value. Theprocessing circuitry may combine the first, second, and third imagesignals to form the derived luminance value by computing a linearcombination of the first, second, and third image signals usingweighting factors.

The processing circuitry may, if desired, perform infinite impulseresponse (IIR) filtering on captured image signals. The processingcircuitry may perform IIR filtering by adjusting the filters applied tothe captured image signals (e.g., the filters as described in connectionwith FIGS. 6-8) based on characteristics of the image signals that arecaptured by image pixels 190. Performing IIR filtering may increase theefficiency with which the processing circuitry processes captured imagesignals.

The processing circuitry may, if desired, perform white balanceoperations on the red, blue, and white image signals. The processingcircuitry may apply a color correction matrix to the white image signalto extract image signals of a different color such as green image signalfrom each white image signal. The processing circuitry may combine thered image signals, blue image signals, green image signals, and whiteimage signals to form a luminance value (e.g., by computing a linearcombination or weighted sum of the red, blue, green, and white imagesignals). The processing circuitry may divide the white image signals bythe luminance value to generate a scaling value. The processingcircuitry may modify the red, green, and blue image signals bymultiplying the red, green, and blue image signals by the scaling value.The scaling value may act as a point filter when operating on the red,green, and blue image signals. If desired, any color image pixels may beused in combination with the white image pixels. If desired, theprocessing circuitry may perform these operations on image signals frommultiple image pixel arrays, image pixel arrays on multiple imagesensors, and/or image signals captured during multiple time frames.

The clear image pixels and associated filtering techniques may beimplemented in a system that also includes a central processing unit,memory, input-output circuitry, and an imaging device that furtherincludes a pixel array, a lens for focusing light onto the pixel array,and a data converting circuit.

The foregoing is merely illustrative of the principles of this inventionwhich can be practiced in other embodiments.

What is claimed is:
 1. An imaging system, comprising: an array of imagesensor pixels, wherein the array of image sensor pixels includes firstimage sensor pixels configured to generate first image signals inresponse to a first frequency range of light, second image sensor pixelsconfigured to generate second image signals in response to a secondfrequency range of light, and third image sensor pixels configured togenerate third image signals in response to a third frequency range oflight; and processing circuitry, wherein the processing circuitry isconfigured to perform filtering operations on the first, second, andthird image signals by: generating a first difference value based on thefirst and third image signals, generating a second difference valuebased on the second and third image signals, applying a filter to thefirst and second difference values to generate filtered differencevalues, and adding the third image signals to the filtered differencevalues.
 2. The imaging system defined in claim 1, wherein the processingcircuitry is configured to perform the filtering operations by, for eachimage sensor pixel of the first image sensor pixels, generating aweighted sum of image signals generated by at least 25 image sensorpixels.
 3. The imaging system defined in claim 1, wherein the processingcircuitry is configured to perform white balance operations on the firstimage signals, the second image signals, and the third image signalsprior to generating the first and second difference values.
 4. Theimaging system defined in claim 3, wherein the processing circuitry isconfigured to apply a color correction matrix to the third image signalsafter generating the filtered difference values, wherein the colorcorrection matrix extracts a fourth image signal of a fourth frequencyrange from the third image signals.
 5. The imaging system defined inclaim 1, wherein the third frequency range is broader than the firstfrequency range and the third frequency range is broader than the secondfrequency range.
 6. The imaging system defined in claim 5, wherein thefiltering operations increase noise correlations between the first,second, and third image signals.
 7. The imaging system defined in claim6, wherein the filtering operations increase and decrease noisefluctuations in the first, second, and third image signals together in acorrelated manner.
 8. The imaging system defined in claim 5, wherein thesecond frequency range is broader than the first frequency range.
 9. Animaging system comprising: first image sensor pixels that gather firstimage signals of a first range of frequencies; second image sensorpixels that gather second image signals of a second range offrequencies; third image sensor pixels that gather third image signalsof a third range of frequencies; and processing circuitry, wherein theprocessing circuitry is configured to generate fourth image signals of afourth range of frequencies using the third image signals, theprocessing circuitry is configured to combine the first image signals,the second image signals, and the fourth image signals to generate afirst luminance value, the processing circuitry is configured to computea second luminance value based on the first image signals, the secondimage signals, and the third image signals, and the processing circuitryis configured to modify the first image signals, the second imagesignals, and the fourth image signals using the first and secondluminance values.
 10. The imaging system defined in claim 9, wherein theprocessing circuitry is configured to compute a scaling value based onthe first and second luminance values, and wherein the processingcircuitry is configured to modify the first image signals, the secondimage signals, and the fourth image signals by multiplying the firstimage signals, the second image signals, and the fourth image signals bythe scaling value.
 11. The imaging system defined in claim 9, whereinthe third range of frequencies is broader than the first range offrequencies and the third range of frequencies is broader than thesecond range of frequencies.
 12. The imaging system defined in claim 11,wherein the third range of frequencies is broader than the fourth rangeof frequencies.
 13. The imaging system defined in claim 11, wherein thesecond range of frequencies is broader than the first range offrequencies.
 14. An imaging system comprising: first image sensor pixelsthat gather first image signals of a first range of frequencies; secondimage sensor pixels that gather second image signals of a second rangeof frequencies; third image sensor pixels that gather third imagesignals of a third range of frequencies that is broader than the firstrange of frequencies and that is broader than the second range offrequencies; and processing circuitry, wherein the processing circuitryis configured to generate a luminance signal by performing a weightedsum of the first, second, and third image signals.
 15. The imagingsystem defined in claim 14, wherein the processing circuitry isconfigured to weight the third image signals more heavily in theweighted sum than the first and second image signals.
 16. The imagingsystem defined in claim 14, wherein the processing circuitry isconfigured to generate a scaling value by dividing the third imagesignals by the luminance value.
 17. The imaging system defined in claim16, wherein the processing circuitry is configured to generate correctedimage signals by multiplying at least the first and second image signalsby the scaling value.
 18. The imaging system defined in claim 17,wherein the processing circuitry is configured to generate fourth imagesignals of a fourth range of frequencies using the third image signalsand wherein the third range of frequencies is broader than the fourthrange of frequencies.
 19. The imaging system defined in claim 14,wherein the processing circuitry is configured to perform filteringoperations on the first, second, and third image signals that increasenoise correlations between the first, second, and third image signals.20. The imaging system defined in claim 19, wherein the filteringoperations increase and decrease noise fluctuations in the first,second, and third image signals together in a correlated manner.